Repository navigation
feat(python): expose table compaction through new_compact_builder - #1323
Conversation
The compaction APIs merged for apache#1247 are reachable from DataFusion's CALL sys.compact but not from the Python binding. Expose the same plan-execute-commit flow as a one-shot builder: - with_partition_filter restricts planning to partitions, combining repeated filters with AND, like the read filters. - with_commit_user overrides the generated commit identity. - execute plans all tasks, executes them and submits one commit; a failed task aborts the files already produced so nothing unreferenced is left behind. No commit is made when no task is planned. Tests cover merging with row preservation, the no-task no-commit case, partition isolation and the commit identity, using write-only tables so the explicit compaction has files to merge. Part of apache#286
| deletion is reported as a ``RuntimeWarning``. | ||
|
|
||
| Warning: This method blocks on a DataFusion runtime. | ||
| Calling this from an active asyncio event loop will result in a panic. |
There was a problem hiding this comment.
[P2] Describe the blocking behavior of execute() in asyncio
An active Python asyncio loop does not enter a Tokio runtime. I ran execute() inside asyncio.run() on a table with two compaction tasks and it completed both tasks. On that path, the synchronous call blocks the Python event-loop thread until it returns. The nested-runtime panic applies when the calling thread is already executing Tokio.
Please document the event-loop blocking behavior here so async callers know when to offload the call to a worker thread.
There was a problem hiding this comment.
Replaced the panic note. The docstring now says execute() blocks the calling thread with the GIL released, so a caller on an asyncio loop should use await asyncio.to_thread(builder.execute).
Co-Authored-By: Claude Code <noreply@anthropic.com>
|
Thank you @JingsongLi |
Purpose
Linked issue: close #1321
Python users can compact a table only by running
CALL sys.compactthroughSQLContext. This addsTable.new_compact_builder(), a thin wrapper over coreCompactBuilderwith the same plan, execute and commit flow as the procedure. Part of #286.Brief change log
CompactBuilder.with_partition_filter(predicate)takes the read-filter dict format, and repeated calls combine with AND.with_commit_user(commit_user)sets the commit user. Both are validated by the core builder when called, as inGlobalIndexDropBuilderandBatchWriteBuilder.execute()plans, runs every task and makes oneCOMPACTcommit. It returns a dict withtasksand the commit's write metrics. Nothing is committed when no task is planned.RuntimeWarningwith the deletion error and then raises the task's error.sys.compactlogs the same message withlog::warn!, but the Python binding has no logger.CompactBuilderandTable.new_compact_builder.Tests
New
test_compact.py:COMPACTsnapshot is published;p=agets new files andp=bis untouched;<commit_user>-compact-<suffix>;ValueErrorand no commit without a local cache; withlocal-cache.enabled=true, compaction makes the rows visible.API and Format
New Python API:
Table.new_compact_builder()andCompactBuilder. No storage format change.Documentation
New Table Maintenance > Compaction section in
docs/src/python-binding.md, and a pointer to it fromdocs/src/compaction.md.Generated-by: Claude Opus